A new tool for evaluating health equity in academic journals; the Diversity Factor

Current methods to evaluate a journal’s impact rely on the downstream citation mapping used to generate the Impact Factor. This approach is a fragile metric prone to being skewed by outlier values and does not speak to a researcher’s contribution to furthering health outcomes for all populations. Therefore, we propose the implementation of a Diversity Factor to fulfill this need and supplement the current metrics. It is composed of four key elements: dataset properties, author country, author gender and departmental affiliation. Due to the significance of each individual element, they should be assessed independently of each other as opposed to being combined into a simplified score to be optimized. Herein, we discuss the necessity of such metrics, provide a framework to build upon, evaluate the current landscape through the lens of each key element and publish the findings on a freely available website that enables further evaluation. The OpenAlex database was used to extract the metadata of all papers published from 2000 until August 2022, and Natural language processing was used to identify individual elements. Features were then displayed individually on a static dashboard developed using TableauPublic, which is available at www.equitablescience.com. In total, 130,721 papers were identified from 7,462 journals where significant underrepresentation of LMIC and Female authors was demonstrated. These findings are pervasive and show no positive correlation with the Journal’s Impact Factor. The systematic collection of the Diversity Factor concept would allow for more detailed analysis, highlight gaps in knowledge, and reflect confidence in the translation of related research. Conversion of this metric to an active pipeline would account for the fact that how we define those most at risk will change over time and quantify responses to particular initiatives. Therefore, continuous measurement of outcomes across groups and those investigating those outcomes will never lose importance. Moving forward, we encourage further revision and improvement by diverse author groups in order to better refine this concept.

reflect confidence in the translation of related research. Conversion of this metric to an active pipeline would account for the fact that how we define those most at risk will change over time and quantify responses to particular initiatives. Therefore, the salient point is that continuous measurement of outcomes across groups and those investigating those outcomes will never lose importance. This statement will be typeset if the manuscript is accepted for publication.
Please review the submission guidelines and the instructions link below for detailed requirements and guidance.
The authors received no specific funding for this work.

Competing Interests
On behalf of all authors, disclose any competing interests that could be perceived to bias this work.
This statement will be typeset if the manuscript is accepted for publication.
Please review the instructions link below and PLOS Global Public Health's competing interests policy to determine what information must be disclosed at submission.
The authors have declared that no competing interests exist.

Data Availability
Before publication, Authors are required to make fully available and without restriction all data underlying their All data have been sourced from open-access sources as described in the methods.
These are freely available from the referenced sources without special privileges.  The characteristics of datasets used to inform clinical decisions are vital as these conclusions 155 will likely have the greatest relevance to the populations studied. This can be further 156 exacerbated as AI has the potential to perpetuate these biases if left unchecked [29]. The 157 authors propose that datasets used in health research should be mapped to highlight 158 geographic and academic areas of data poverty, expose underlying knowledge gaps, and 159 draw attention to imbalanced datasets [30]. These include gender imbalance, race-ethnicity, 160 language preference, age, and geography. The current impact factor rewards citations equally 161 but, determining who has read, utilized, and has been 'impacted' by the research is not as 162 simple as implied. Most data being analyzed to guide healthcare is derived from a few centers, 163 almost exclusively based in High-Income Countries [31]. As such, increasing the global impact 164 of research is vital, and the increasing adoption of technology has created the potential to 165 democratize health research. More effort should then be placed on increasing the 166 diversification of the data pool used to design clinical guidelines and develop tools that provide 167 beneficial outcomes for all, and not just a select few countries. 168 8

Author Country 170
Author's previous experiences, surrounding culture, and the associated team will significantly 171 shape projects. Thus, when considering how to evaluate an author group, it is important to 172 heavily weigh those with increased diversity, as these groups will better reflect the populations 173 in question as well as having a wider range of problem-solving abilities and perspectives. 174 Evaluating the spread in the country of affiliations within studies can speak to the cognitive 175 diversity of the teams and the likelihood of methodology and results transferring to that area. 176 The authors designed the study methodology, conducted the analysis, interpreted the findings 177 and presented these in an organized manner. Throughout these stages, biases can be 178 introduced by influencing selection strategies, modes of analyses and presentation of results. 179 Authors working in one country who analyze datasets from another have inherent limitations 180 due to an incomplete understanding of the context and culture of the studied subjects. 181 Including diverse perspectives can maximize the scope for identifying potential biases and 182 ensuring the results produced are applicable to multiple locations. Notably, historical racial 183 prejudices have resulted in disparities in clinical outcomes between demographics and 184 including a variety of ethnicities would improve safeguarding against introducing similar biases 185 [32]. It is commonplace for knowledge to centralize, with intellectual centers producing 186 multitudes of research. These 'Ivory Towers' often overrepresent a particular demographic 187 that is inconsistent with the experience or backgrounds of those most burdened by disease. 188 Increasing diversity within author groups, especially within institutions, can help combat this 189 resulting homogeneity of thought. 190 191 3. Author Gender 192 Diversity is more than just increasing the number of ethnicities within the author group. 193 Traditionally, academia has been a male-dominated field, well-documented across multiple 194 fields [33,34]. However within recent years, this trend has been shifting -more and more 195 women are joining the field. However, gender-parity has not yet been reached and given the 196 current trends will take several more years unless there are active, intentional changes. In this 9 study, an algorithm trained to identify 'gender' as Male or Female was utilized. However, as 198 further research is done to refine this proposal, expanding this definition to include other 199 genders and the distinction between sex and gender would be ideal. Most health research has traditionally been conducted in a few institutions with the 203 necessary funding and access to data. Considering the centralization of knowledge and 204 the overrepresentation of certain demographics found in these ivory towers, there has 205 been a recent shift to increase engagement with local stakeholders and increase patient 206 public involvement. Further, healthcare is increasingly becoming a multi-disciplinary field 207 as a result of the recognition that socioeconomic factors play significant roles in health 208 outcomes. As such, multidisciplinary teams play an important role in bridging professional 209 boundaries and breaking down the barriers of competing cultural and organizational 210 differences will improve the translational gap in medical devices and root academic work 211 in deployable applications. There are currently divides between clinical, academic, and 212 commercial research that often leaves everyone feeling that data is out of reach. Bridging 213 the gap between commercial and non-commercial fields will improve the translational gap 214 in medical devices and root academic work in deployable applications. Understanding the 215 current breakdown of institutional affiliation, whether it is academic or commercial, is 216 necessary to see if academia is responding to this multidisciplinary call. Expansion of this 217 definition to include the composition of expertise will also be vital, for example, the 218 interaction of machine learning engineers and social scientists in the field of AI Enriched metadata is produced using fine-tuned Natural Language Processing models 238 (BERT-Pubmed) for research classification and entity extraction, as described elsewhere [1]. 239 Affiliation strings derived from this process were then parsed for information on departmental 240 affiliation that were then categorized into commercial and academic organizations. 241

242
We identified the author's gender using several APIs that demonstrate state-of-the-art 243 performance in validation studies on non-English names, including Gender-API and 244 Genderize [40,41]. Gender matching was conducted using the first name and affiliation 245 country, with 84% of entries matched. The female: male author ratio was calculated for each 246 paper, and then a mean was calculated for each journal and time period. 247

248
Features are displayed individually, and a hypothesis of aggregation is discussed below. 249 Descriptive statistics for each of the three included features is displayed in a subset of eight 250 journals that cover the broadest ranges of speciality, Impact Factor, and traditional prestige. The next steps for implementing a diversity factor would include the availability of dataset 302 characteristics, detailed funding sources, patents and downstream policy impact, and citation 303 mapping. This would allow for a better understanding of who is impacted and who is causing 304 the impact. Additionally, refining the use of author country to a place of birth or time spent in 305 a country would account for those in LMICs who emigrate to other institutions, which is not 306 uncommon. Further, the improvement of gender data or NLP algorithms to account for author 307 gender compared to sex would be another step forward. The operationalisation of this tool 308 would rely on interest from journals and researchers in collaborating to permit this data to be 309 published on each journal's website and centrally for evaluation. The heterogeneity in the 310 findings between these four diversity metrics and between journals likely means that these 311 four values should not be combined into one 'diversity value'. Instead, they should be 312 evaluated and compared individually as seen in Figure  allow for more detailed analysis to be performed, highlight gaps in knowledge, and reflect 318 confidence in the translation of related research. This is particularly true at the health policy 319 level, where it is known that the social determinants of health vary greatly between countries; 320 therefore, clinical decisions and public health decisions should be made based on information 321 more representative of these populations. It is important to acknowledge that there is not one 322 type of bias, nor one group affected by bias, but many types of bias and many groups that can 323 be biased against. Diversity is not a box-ticking exercise but an essential safeguard against 324 potential biases, especially in countries with greater ethnicity, cultural diversity and particular socio-325 demographic characteristics. Promotion of these features would encourage thoughtful discourse 326 on how study designs and data characteristics can affect different groups so that researchers 327 and organizations can build the right team for the specific project and its risks. Furthermore, 328 14 integration of the Diversity Factor can encourage collaboration with LMICs that could reshape 329 the knowledge landscape through the dissemination of work partnered with LMICs. 330

331
Our goal was to evaluate the current diversity factor landscape to the highest degree 332 permissible by the availability of data and the current standards of metadata. Data set 333 characteristics were not widely available, which did not permit the evaluation of this important 334 feature. However, we have provided the tools to implement this feature if made available in 335 the future. Gender was determined using NLP, which has demonstrated good performance 336 across countries; however, the ground truth is not available for these studies and so cannot 337 be confirmed. SCImago Journal Ranking is used in place of traditional impact factor due to 338 the open source nature and availability of the feature, there is a similarity between the two, 339 but we also acknowledge differences. We recognise these tools are imperfect, but we hope   to being combined into a simplified score to be optimized. Herein, we discuss the necessity of 75 such metrics, provide a framework to build upon, evaluate the current landscape through the 76 lens of each key element Therefore, we developed the Diversity Factor to fulfil this need and 77 comprised four key elements: dataset properties, author country, author gender and 78 departmental affiliation. Herein, we develop the methodology to reproducibly calculate these 79 elements, evaluate the current landscape and publish the findings on a freely available website 80 that enables further evaluation. 81 The OpenAlex database was used to extract the metadata of all papers published from 2000 82 until August 2022, and Natural language processing was used to identify individual elements. 83 Features were then displayed individually on a static dashboard developed using 84 TableauPublic, which is available at www.equitablescience.com. 85 In total, 130,721 papers were identified from 7,462 journals where significant 86 underrepresentation of LMIC and Female authors was demonstrated. These findings are 87 pervasive and show no positive correlation with the Journal's Impact Factor. 88 The systematic collection of the Diversity Factor concept would allow for more detailed 89 analysis, highlight gaps in knowledge, and reflect confidence in the translation of related 90 research. Conversion of this metric to an active pipeline would account for the fact that how 91 we define those most at risk will change over time and quantify responses to particular 92 initiatives. Therefore, continuous measurement of outcomes across groups and those 93 5 investigating those outcomes will never lose importance. Moving forward, we encourage 94 further revision and improvement by diverse author groups in order to better refine this 95 concept. These four elements should each be seen as a new way to rank journals rather than 96 a simplified score to be optimized. The systematic collection of these features would allow for 97 more detailed analysis, highlight gaps in knowledge, and reflect confidence in the translation 98 of related research. Conversion of this metric to an active pipeline would account for the fact 99 that how we define those most at risk will change over time and quantify responses to 100 particular initiatives. Therefore, the salient point is that continuous measurement of outcomes 101 across groups and those investigating those outcomes will never lose importance. 102 103 Background 104 The last decade has seen our capacity to store, analyse and distribute health data grow 105 exponentially, especially with the growing use of artificial intelligence (AI), yet healthcare has 106 tried and failed to implement it in a successful manner. The current AI landscape is ever-107 expanding and many of the current models are either still in the prototype stage [1,2]  Oversight. These are explored in Table 1 along with guiding questions that describe the 172 feature's respective characteristics. It is important to note that this is merely a proposal and 8 should serve as a foundation to build upon. Additionally, this proposed Diversity Factor should 174 serve as a supplement to the currently used metrics and not as a replacement. Scoring of this 175 concept is also left for future revisions. For this paper, each key element is reviewed 176 independent of the others.The Diversity Factor aims to propose an alternative means of 177 tracking accurate and reliable contributions to health research aligned with the impact on the 178 global community or population. It offers an approach to facilitate scientific excellence that is 179 unbiased, representative and impactful. Important factors that should be considered in 180 evaluating the literature produced from a given journal are explored in Table 1 Author's previous experiences, surrounding culture, and the associated team will significantly 225 shape projects. Thus, when considering how to evaluate an author group, it is important to 226 heavily weigh those with increased diversity, as these groups will better reflect the populations 227 in question as well as having a wider range of problem-solving abilities and perspectives. 228 Evaluating the spread in the country of affiliations within studies can speak to the cognitive 229 diversity of the teams and the likelihood of methodology and results transferring to that area. 230 The authors designed the study methodology, conducted the analysis, interpreted the findings 231 and presented these in an organized manner. Throughout these stages, biases can be 232 introduced by influencing selection strategies, modes of analyses and presentation of results. 233 Authors working in one country who analyze datasets from another have inherent limitations 234 due to an incomplete understanding of the context and culture of the studied subjects. Diversity is more than just increasing the number of ethnicities within the author group. 252 Traditionally, academia has been a male-dominated field, well-documented across multiple 253 fields [33,34][33,34]. However within recent years, this trend has been shifting -more and 254 more women are joining the field. However, gender-parity has not yet been reached and given 255 the current trends will take several more years unless there are active, intentional changes. In 256 this study, an algorithm trained to identify 'gender' as Male or Female was utilized. However, 257  Enriched metadata is produced using fine-tuned Natural Language Processing models 309 (BERT-Pubmed) for research classification and entity extraction, as described elsewhere 310 representation for female authors, it is unclear how many journals, if any, will reach gender-337 parity in the next five years given the rate of improvement. Those most at risk of being 'left 338 behind' are primarily from low-income countries though some countries such as Japan    Thank you for the opportunity to submit a revision of our paper on the Diversity Factor. We are pleased that you have been very supportive of its publication in PLOS Global Public Health.
We have made the revisions as suggested and are grateful for your time and critical insight, which has improved the study further.
Thank you for handling our submission, and we look forward to hearing from the journal soon.

Yours sincerely
We thank the Editor and Reviewer for their constructive thoughts on this piece. The paper has been rewritten and restructured to strengthen the argument as suggested. Concepts were expanded upon as necessary.
[Issue 1]: The manuscript requires an overall strengthening of the argument. For example, concepts such as commercial and non-commercial fields, ITA, etc., can be better explained and expanded on. The paper has been rewritten and restructured to strengthen the argument as suggested. Concepts were expanded upon as necessary. Changes are highlighted in the document for clarity.
[Issue 2]: The literature review needs to be expanded to include more recent and relevant studies and a more comprehensive analysis of the existing body of research. Literature review was expanded as requested. 8 additional references were added.
[Issue 3]: The connection between author diversity and research impact needs to be clearly defined.The manuscript would strongly benefit from a more robust and nuanced explanation of this connection. The entire paper was rewritten to explain this connection better throughout. This is best seen in Lines 120-124.
[Issue 4]: The connection of the diversity factor and the field of digital health needs to be described in more detail as it currently leads to more questions than answers. Please review this by either highlighting this connection or by avoiding using the term digital health, as it entails a specific definition of a particular field. This is a very important clarification to make. We have elaborated on this throughout the paper to highlight the necessity of the importance of diversity factor; the term digital health was also removed.
[Issue 5]: Line 158, cognitive diversity does not necessarily convey the diversity of expertise, lived experiences, or points of view and I encourage you to revisit this phrase/concept.